Papers with regularization techniques
Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing methods to learn mappings in the embedding space of text autoencoders are limited to a single-vector embeddment, which limits how much information can be retained. |
| Approach: | They propose a method to learn mappings in the embedding space of an autoencoder by extending it to Bag-of-Vectors Autoencodeurs (BoV-AEs) this allows to encode and reconstruct much longer texts than standard autoencodings . |
| Outcome: | The proposed method performs better than a standard autoencoder on unsupervised sentiment transfer. |
Improving Few-Shot Cross-Domain Named Entity Recognition by Instruction Tuning a Word-Embedding based Retrieval Augmented Large Language Model (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to named entity recognition are domain specific and require a domain specific architecture. |
| Approach: | They propose a retrieval augmented large language model for Named Entity Recognition . the model uses word-embedding over sentence-level embedding to fine tune . |
| Outcome: | The proposed model outperforms existing models on the CrossNER dataset. |
Investigating the Working of Text Classifiers (C18-1)
Copied to clipboard
| Challenge: | Text classification is one of the most widely studied tasks in natural language processing. |
| Approach: | They propose to use large multilayer neural network models to compose meaning of sentences . they propose to disincentivize focusing on key lexicons to improve classification accuracy . |
| Outcome: | The proposed models learn to compose the meaning of the sentences or focus on key lexicons for classifying the document. |
Domain adaptation for part-of-speech tagging of noisy user-generated text (N19-1)
Copied to clipboard
| Challenge: | Existing POS taggers for canonical German text achieve good results around 97% accuracy, but when applying these trained models to out-of-domain data the performance decreases drastically. |
| Approach: | They propose a neural network that trains an out-of-domain model on a large newswire corpus and transfers those weights by using them as a prior for a model trained on the target domain. |
| Outcome: | The proposed model achieves a tagging accuracy of slightly over 90%, improving on the previous state of the art for this task. |
Exploiting Document Structures and Cluster Consistencies for Event Coreference Resolution (2021.acl-long)
Copied to clipboard
| Challenge: | Existing deep learning models for event coreference resolution are limited in that they cannot exploit important interactions between relevant objects for ECR. |
| Approach: | They propose a deep learning model that groups coreferent event mentions into the same clusters . they use document structures to capture relevant objects for ECR . |
| Outcome: | The proposed model achieves state-of-the-art on two benchmark datasets. |
Rethinking Complex Neural Network Architectures for Document Classification (N19-1)
Copied to clipboard
| Challenge: | Neural network models for many NLP tasks have grown increasingly complex in recent years . authors of recent papers question the necessity of such architectures and find them quite effective . |
| Approach: | They propose to use regularization techniques borrowed from language modeling to improve model accuracy . they find that a simple biLSTM architecture with appropriate regularization yields competitive results . |
| Outcome: | a simple biLSTM model outperforms the state-of-the-art on four benchmark datasets . authors say that improvements are not real, but are attributed to mundane reasons . |
Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods (2023.findings-acl)
Copied to clipboard
| Challenge: | A popular approach to unveiling the black box of neural NLP models is to leverage saliency methods, which assign scalar importance scores to each input component. |
| Approach: | They propose to use saliency methods to evaluate whether an explanation is faithful and argue that Pearson-r is a better-suited alternative to rank correlation. |
| Outcome: | The proposed methods exhibit weak rank correlations even when applied to the same model instance and advocated for alternative diagnostic methods. |
Bridging the Empirical-Theoretical Gap in Neural Network Formal Language Learning Using Minimum Description Length (2024.acl-long)
Copied to clipboard
| Challenge: | Neural networks offer good approximation to many tasks but fail to reach perfect generalization. |
| Approach: | They propose to use a formal language to test whether a theoretically correct solution is not an optimum of commonly used objectives. |
| Outcome: | The proposed model fails to reach the theoretically correct solution even with regularization techniques. |
IR2: Information Regularization for Information Retrieval (2024.lrec-main)
Copied to clipboard
| Challenge: | Effective information retrieval (IR) in settings with limited training data remains a challenging task. |
| Approach: | They propose a technique for reducing overfitting during synthetic data generation . they use DORIS-MAE, ArguAna, and WhatsThatBook as examples . |
| Outcome: | The proposed technique outperforms previous methods and reduces cost by 50% on three recent IR tasks characterized by complex queries. |
Lifelong Knowledge Editing requires Better Regularization (2025.findings-emnlp)
Copied to clipboard
Akshat Gupta, Phudish Prateepamornkul, Maochuan Lu, Ahmed Alaa, Thomas Hartvigsen, Gopala Anumanchipalli
| Challenge: | Knowledge editing is a promising way to improve factuality in large language models, but recent studies have shown significant model degradation during sequential editing. |
| Approach: | They formalize locate-then-edit methods as a two-step fine-tuning process . they show that model degradation occurs due to over-optimization of internal activations . |
| Outcome: | The proposed methods reduce time and improve factuality by 42-61%. |